Databricks Runtime Versions
Introduction
Databricks Runtime (DBR) is a set of core software components such as Spark, libraries, drivers that run on the compute clusters managed by Databricks. Databricks releases updated versions of its Runtime to introduce new features, enhance performance, and improve security.
Using the appropriate Databricks Runtime Version is necessary in Data Pipeline Studio because it defines the core engine that powers the data pipelines. It guarantees compatibility with the newest features, performance improvements, and security patches, all of which are crucial for the dependable, efficient, and secure operation of data pipelines. Runtime version grants access to the latest libraries and tools, enabling more advanced processing capabilities within the pipeline.
Why are Databricks Runtime versions important?
Each Databricks Runtime (DBR) version includes specific capabilities, supported libraries, and behavioral nuances that can directly affect how your pipelines run. Keeping up with runtime versions is critical because they influence the performance, features, and security of your data processing and analytics workloads.
-
Access to new features and integrations: Each release introduces updated tools, libraries, and ecosystem integrations that enable more advanced and efficient data processing.
-
Performance improvements: Version updates often include optimizations that help jobs run faster and consume fewer resources.
-
Stability and reliability: Updates resolve bugs and enhance workload robustness.
-
Security and compliance: New versions patch vulnerabilities and prevent issues caused by outdated components.
Why do Databricks Runtime versions differ?
Databricks provides multiple runtime versions, each optimized with specific enhancements, updates, and fixes to support different workload types.
-
Performance Enhancements
Newer runtime versions deliver better memory management, faster execution, and improved resource efficiency.
-
New Features & Capabilities
As new tools and libraries are developed, runtime updates include them so users can leverage the latest technologies.
-
Security Updates
Regular releases strengthen protection against emerging security threats and help meet evolving compliance requirements.
-
Library and Tool Updates
Core components such as Apache Spark and Delta Lake are frequently updated to incorporate bug fixes, new functionality, and performance improvements.
-
Ecosystem Compatibility
Runtime updates ensure seamless integration with major cloud platforms (AWS, Azure) and the broader big data ecosystem.
Databricks Runtime Versions in DPS
The following table helps you to identify the Databricks Runtime (DBR) versions that are supported or required for use in different types of pipelines in the Calibo Accelerate platform.
|
Job Type |
Databricks Runtime Version |
|||||
|---|---|---|---|---|---|---|
|
11.3 LTS |
12.2 LTS |
13.3 LTS |
14.3 LTS |
15.4 LTS |
||
|
Data Integration |
Templatized jobs |
|
Yes |
Yes |
Yes |
Yes |
|
Custom jobs |
Yes | Yes | Yes | Yes | Yes | |
|
Data Transformation
|
Templatized jobs |
Yes | Yes | Yes | Yes | Yes |
|
Custom jobs |
Yes | Yes | Yes | Yes | Yes | |
|
Data Quality
|
Data Profiler |
|
Yes | Yes |
|
|
|
Data Validator |
|
Yes | Yes |
|
|
|
|
Data Analyzer |
|
Yes | Yes |
|
|
|
|
Data Issue Resolver |
|
|
|
Yes | Yes | |